fix(ducklake): GC leaked inline tables to stop multi-GB RSS growth - #767
Merged
Conversation
DuckLake creates a ducklake_inlined_data_<table_id>_<schema_version> table per schema version, schema_version bumps on every DDL (incl. our per-table SET PARTITIONED BY / SET SORTED BY), and no maintenance path ever DROPs them (flush only DELETEs rows, expire only drops registry rows, cleanup ignores them). They accumulate forever and the extension rebuilds an in-memory stats map (DuckLakeCatalog::ConstructStatsMap) over all of them on every catalog refresh, growing RSS to multiple GB. Upstream bug duckdb/ducklake#1065. - Startup GC (GCOrphanInlineTables): before ATTACH (exclusive sqlite access, same window as EnableSQLiteWALMode) drop every EMPTY ducklake_inlined_data_* table and its ducklake_inlined_data_tables registry row. Empty => rows already flushed to Parquet, so lossless. Best-effort, non-fatal. - Stop the churn: SET PARTITIONED BY / SET SORTED BY now run only when the HEP/OTLP table is first created (gated on information_schema.tables), not on every startup (each re-issue bumped schema_version and spawned another leaked inline table per restart). Bump 11.0.234 -> 11.0.235.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
DuckLake leaks
ducklake_inlined_data_*physical tables — upstream bug duckdb/ducklake#1065:ducklake_inlined_data_<table_id>_<schema_version>table per schema version.schema_versionbumps on every DDL — including theSET PARTITIONED BY/SET SORTED BYwe issue for eachhep_proto_*andotlp_*table on every startup.flush_inlined_dataonlyDELETEs their rows,expire_snapshotsonly removes registry rows,cleanup_old_filesignores them.They accumulate forever, and the DuckLake extension rebuilds an in-memory stats map (
DuckLakeCatalog::ConstructStatsMap) over all of them on every catalog refresh → RSS climbs without bound even with a tiny catalog and few Parquet files.This is the same defect diagnosed and fixed in the sibling ingest service (heap profile pinned ~2.5 GB to
ConstructStatsMapfed by ~870 leaked inline tables; dropping the empty ones took RSS from 4.1 GB back to ~300 MB).Fix
GCOrphanInlineTables). BeforeATTACH— while the sqlite catalog is still exclusively ours, the same windowEnableSQLiteWALModealready uses — drop every emptyducklake_inlined_data_*table and delete itsducklake_inlined_data_tablesregistry row. Empty ⇒ rows already flushed to Parquet, so the cleanup is lossless. Best-effort, non-fatal. Non-empty inline tables (un-flushed legacy rows) are left untouched.SET PARTITIONED BY/SET SORTED BYnow run only when the table is first created (gated oninformation_schema.tables), for both HEP (tables.go) and OTLP (otlp_storage.go) tables — not on every boot. Re-issuing them bumpedschema_versioneach restart and spawned a fresh leaked inline table per table per restart.Bump
11.0.234→11.0.235.Notes
.sqlite,DROPthe emptyducklake_inlined_data_*tables + their registry rows,VACUUM, restart.Test plan
ducklake_inlined_data_*backlog; confirm the "DuckLake inline GC: dropped N ..." log and thatducklake_inlined_data_tablesshrinks to the non-empty set.hep_proto_*andotlp_*tables are still createdPARTITIONED BY (date)/SORTED BY (timestamp ASC).